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FOREWORD 


This  training  system  effectiveness  evalaution  effort  was  conducted  in  support  of 
Navy  Decision  Coordinating  Paper  (NDCP)  W0784-PN,  of  6  July  1977,  Work  Unit  No. 
N001985WR6322W  under  the  sponsorship  of  Naval  Air  Systems  Command  (NAVAFR 
APC205-ON).  It  was  carried  out  with  the  consent  of  the  Naval  Air  Maintenance  Training 
Group  (NAMTRAGRU),  Memphis  and  the  active  participation  of  NAMTRAGRU  Detach¬ 
ment  1069,  Naval  Air  Station,  North  Island,  California. 

NAVAIR  in  cooperation  with  the  NAMTRAGRU  Memphis  selected  the  SH-3  helicop¬ 
ter  as  a  testbed  for  evaluating  an  application  of  generalized  maintenance  trainer/simu¬ 
lator  (GMTS)  technology  in  aviation  maintenance  training  courses.  The  GMTS  system  to 
be  evaluated  was  developed  by  the  Navy  Personnel  Research  and  Development  Center  and 
fabricated  by  the  Cubic  Corporation.  Two  prototype  systems,  referred  to  as  the  bladefold 
electro-mechanical  training  systems  (BEMTSs),  were  delivered  to  the  NAMTRAGRU 
Detachment  1069  in  June  1985.  Testing  began  in  November  1985  and  was  completed  in 
December  1986. 

Appreciation  is  expressed  for  the  outstanding  cooperation  of  NAMTRAGRU  Detach¬ 
ment  1069  Chief  Petty  Officer  in  Charge  AFCM  Leroy  Delaney,  AECS  Harold  Cochran, 
and  AMCS  Michael  Fortney  for  supporting  the  scheduling  and  administration  of  the 
evaluation.  Appreciation  is  also  expressed  to  the  instructor  staff,  which  included  AM{H)1 
Terry  Watson,  AEl  Hugh  Bennett,  AM(H)1  Jose  Caspar,  AH(H)1  Charles  Worley  III,  and 
AEl  Wanda  Gunderson,  for  graciously  accepting  the  additional  work  and  displaying  the 
professionalism  and  dedication  needed  to  complete  this  research. 


B.  E.  BACON  J.  S.  McMICHAEL 

Captain,  U.S.  Navy  Technical  Director 

Commanding  Officer 


SUMMARY 


Problem  and  Background 

The  number  of  training  systems  available  to  support  hands-on  training  of  critical 
maintenance  skills  are  often  too  few  to  allow  individual  trainees  sufficient  time  on 
training  tasks.  While  important  for  component  task  consolidation,  the  exclusive  use  of 
actual  equipment  or  high  fidelity  simulation  devices  for  teaching  systems  maintenance 
bears  inherent  cost  and  pedagogical  limitations  that  can,  and  often  do,  affect  the  quality 
of  available  training.  Recent  advances  in  training  systems  technology  make  it  possible  to 
separately  develop  and  exercise  the  cognitive  skill  involved  in  systems  diagnosis  and 
troubleshooting  training.  A  generalized  maintenance  training  system  (GMTS)  was 
developed  to  minimize  the  effects  of  limited  training  hardware  availability  and  to 
increase  training  effectiveness  in  existing  laboratory  maintenance  training.  The  Navy 
Personnel  Research  and  Development  Center  was  tasked  by  the  Naval  Air  Systems 
Command  (NAVAIR  APC205-ON)  to  evaluate  the  new  technology  in  connection  with  the 
SH-3  helicopter  electro-mechanical  bladefold  system  in  the  Naval  Aviation  Maintenance 
Training  Group  (NAMTRAGRU)  detachment  environment. 

Objectives 

The  objectives  of  this  effort  were  to  (1)  determine  the  effectiveness  of  the  bladefold 
electro-mechanical  training  system  (BEMTS)  in  developing  organizational-level  main¬ 
tenance  skills  of  aviation  structural  mechanic,  hydraulic  (AM{H))  and  aviation  electrician 
(AE)  trainees  assigned  to  the  bladefold  system  portion  of  the  NAMTRAGRU  Detachment 
1069  SH-3  helicopter  hydraulic  and  electrical  systems  training  courses,  (2)  to  establish  a 
performance  baseline  for  future  testing  of  intelligent  tutoring  system  techniques  in 
connection  with  BEMTS  freeplay  simulation  capabilities,  and  (3)  to  evaluate  the  suitability 
of  BEMTS  to  the  NAMTRAGRU  training  environment. 

Approach 

Two  identical  BEMTS  were  installed  at  the  NAMTRAGRU  Detachment  1069,  Naval 
Air  Station,  North  Island,  CA  to  supplement  training  in  the  SH-3  helicopter  bladefold 
system  portion  of  the  airframe  and  hydraulic  systems  and  the  electrical  systems 
organizational-level  maintenance  courses.  For  the  evaluation,  trainees  attending  the  two 
courses  were  divided  into  balanced  groups  based  on  years  of  experience  in  aviation 
maintenance  and  relative  class  standing.  Trainees  in  each  experimental  group  used  the 
BEMTS  as  a  supplement  to  their  training,  while  trainees  in  each  control  group  spent 
equivalent  time  in  independent  study  reviewing  class  notes,  system  schematics,  and 
troubleshooting  manuals.  A  total  of  97  trainees,  71  AM(H)s  and  27  AEs,  participated  in 
the  evaluation. 

Results 


The  evaluation  results,  though  statistically  inconclusive,  indicate  that  both  the 
experimental  groups  received  higher  final  grades  on  written  and  performance  test,  took 
less  time  to  troubleshoot  malfunctions  on  the  SH-3  composite  trainer,  and  made  fewer 
diagnostic  and  troubleshooting  errors  than  did  their  control  group  counterparts.  The 
results  of  user  acceptance  and  suitability  interviews  and  questionnaires  indicate  that  the 
instructors  and  trainees  generally  liked  the  technology  and  considered  it  well  suited  for 
bladefold  system  maintenance  training.  Intermittent  BEMTS  hardware  and  software 
problems  that  occurred  early  in  the  evaluation  effort  caused  initial  concern  about  overall 
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training  system  availability.  However,  most  hardware  and  software  problems  were 
resolved  and  did  not  adversely  affect  the  on-going  evaluation. 

Conclusions 


The  use  of  BEMTS  can  be  expected  to  (1)  reduce  mean  troubleshooting  time  and  mean 
errors  to  problem  solution  for  both  AM(H)  and  AE  technician  trainees  performing 
exercises  on  the  SH-3  helicopter  composite  trainer,  and  (2)  increase  mean  test  scores  on 
written  tests  of  bladefold  systems  knowledge  for  both  AM{H)  and  AE  technician  trainees. 
Further,  the  data  collected  in  this  effort  established  an  adequate  performance  base  from 
which  to  measure  and  assess  the  effects  of  intelligent  tutoring  techniques. 

Other  conclusions  addressing  the  continued  use  of  BEMTS  in  NAMTRAGRU  Detach¬ 
ments  are:  (1)  the  BEMTS  simulation  was  not  appropriately  configured  to  provide  the 
visual  and  auditory  symptoms  that  AM(H)  technicians  use  to  diagnose  and  troubleshoot 
hydraulic  system  failures,  (2)  instructor  requirements  associated  with  BEMTS  use  remain 
to  be  determined,  (3)  developing  new  simulation  and  lesson  materials  and  revising  existing 
materials  are  not  within  the  present  or  projected  capabilities  of  NAMTRAGRU  instruc¬ 
tors,  (4)  and  BEMTS  configuration  is  not  consistent  with  recent  trends  toward  standardiza¬ 
tion  on  PC  compatible  hardware. 

Recommendations 


1.  Given  the  positive  direction  of  the  results  of  using  BEMTS  as  a  supplement  to  the 
present  hands-on  equipment  training  approach  in  SH-3  bladefold  system  training,  NAVAIR 
and  NAMTRAGRU  should  continue  to  support  a  policy  to  refine  and  institutionalize  the 
technology  base  established  in  the  BEMTS. 

2.  The  BEMTS  hardware  components  that  are  not  commercially  available  should  be 
replaced  with  PC  compatible  devices.  Future  testing  should  include  an  investigation  of 
BEMTS  software  transportability  to  other  aviation  maintenance  training  courses  where 
similar  low-cost  training  systems  are  being  planned  or  used. 

3.  The  BEMTS  simulation  and  tutorial  lesson  data  bases  should  be  expanded  to 
include  other  applicable  portions  of  the  SH-3  helicopter  maintenance  training  courses  and 
additional  tests  should  be  conducted  to  measure  the  actual  impact  of  BEMTS  on  instructor 
workloads  under  full  curriculum  implementation  conditions. 

4.  The  display  techniques  incorporated  in  the  BEMTS  freeplay  and  tutorial  para¬ 
digms  should  be  expanded  to  implement  the  visual  and  auditory  cues  AM(H)  technicians 
use  to  diagnose  hydraulic  systems  failures  and  tests  should  be  designed  to  measure  the 
applicability  and  effectiveness  of  these  simulation  and  training  enhancements. 

5.  Responsibility  for  BEMTS  simulation  and  lesson  authoring  should  not  be  assigned 
to  the  user  command  unless  such  tasking  includes  appropriate  training  and  corresponding 
augmentations  in  instructor  staffing  plans. 
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INTRODUCTION 


Problem 


The  number  of  training  systems  available  to  support  hands-on  training  of  critical 
maintenance  skills  are  often  too  few  to  allow  individual  trainees  sufficient  time  on 
training  tasks.  While  important  for  component  task  consolidation,  the  exclusive  use  of 
actual  equipment  on  high  fidelity  simulation  devices  for  teaching  systems  maintenance 
bears  inherent  physical  and  pedagogical  limitations.  These  include: 

1.  Fault  insertion  capabilities  are  often  limited  and  cannot  adequately  support  a 
comprehensive  systems  troubleshooting  training  program. 

2.  Training  conditions  may  be  unsafe  for  novice  technicians. 

3.  Less  training  equipment  than  required  to  meet  actual  training  needs  is  often 
available. 

4.  Training  system  upgrades  and  modifications  are  typically  behind  fleet  installa¬ 
tions. 

5.  Trainee  performance  is  often  difficult  to  evaluate  objectively. 

6.  When  more  than  one  trainee  is  assigned  the  same  performance  task,  individual 
performance  deficiencies  are  not  easily  measured  or  corrected. 

7.  Training  often  focuses  on  repair  tasks  rather  than  on  the  functional  relationships 
that  lead  to  an  in-depth  understanding  of  the  hardware  system. 

These  limitations  can  significantly  affect  the  quality  of  available  training  and  the 
ability  of  the  technical  training  commands  to  provide  qualified  first-  and  second-term 
technicians  to  the  fleet. 

Background 

Recent  advances  in  training  systems  technology  make  it  possible  to  separately 
develop  and  exercise  the  cognitive  skills  involved  in  system  diagnosis  and  troubleshooting. 
These  skills  involve  such  tasks  as  formulating  hypotheses  about  a  given  system  failure, 
selecting  and  performing  appropriate  system  tests,  and  making  inferences  from  the  results 
of  each  test.  The  ability  to  exercise  these  skills  as  well  as  monitor  and  track  student 
performance  makes  this  technology  even  more  attractive  for  future  applications  of 
intelligent  tutoring  systems. 

The  bladefold  electro-mechanical  training  system  (BEMTS)  is  a  computer-based 
interactive  simulation  and  training  device.  It  was  developed  to  eliminate  some  of  the 
physical  and  pedagogical  limitations  inherent  in  the  exclusive  use  of  actual  equipment 
trainers  in  maintenance  oriented  training  programs.  In  its  present  configuration,  BEMTS 
consists  primarily  of  off-the-shelf  commercial  hardware  incorporating  a  microcomputer,  a 
videodisc  player,  and  two  CRT  monitors  (a  high  resolution  video/graphics  color  monitor 
and  a  monochrome  computer  text  monitor).  Each  monitor  has  touch-sensitive  panels  for 
user  input  and  a  portable  keyboard  for  instructor  input  and  lesson  authoring.  The 
underlying  simulation  and  training  concept  involves  the  use  of  static  photographic  images 
(scenes)  stored  on  a  videodisc  to  present  visual  representations  of  the  target  system 
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(bladefold)  and  of  the  test  equipment  used  for  system  diagnosis.  Individual  scenes  include 
the  system  and  component-level  views  needed  to  access  and  manipulate  system  controls 
and  to  acquire  information  about  the  present  state  of  the  system.  BEMT5  can  display 
computer  generated  graphics  and  overwrite  graphics  on  individual  video  images. 

BEMTS  software  supports  both  structured  learning  and  freeplay  troubleshooting 
exercises.  In  the  SH-3  helicopter  training  program,  BEMTS  provides  instruction  on 
electrical  and  hydraulic  components  of  the  bladefold  system  and  also  a  troubleshooting 
practice  environment  that  supplements  the  hands-on  training  performed  on  the  bladefold 
system  composite  trainer.  The  structured  training  scenario  employs  a  standard  inter¬ 
active  tutorial  format  focusing  on  component  nomenclature,  location,  and  related  course 
declarative-knowledge  objectives.  The  freeplay  scenario  has  two  training  options.  The 
first  is  a  semistructured  procedure-following  paradigm  in  which  the  trainee  performs 
serial  tasks  in  accordance  with  predefined  steps  of  procedure.  The  second  is  a  freeplay 
troubleshooting  paradigm  in  which  the  trainee  is  presented  with  indications  of  a  system 
failure  (gripe)  and  is  required  to  isolate  the  failure  to  a  faulty  component  and  to  restore 
the  system  to  normal  operations.  For  all  lessons,  the  BEMTS  retains  a  record  of  student 
performance  that  can  be  accessed  and  evaluated  by  course  instructors. 

The  BEMTS  is  intended  to  develop  organizational  (O)  level  and  intermediate  (I)  level 
maintenance  skills  by  simulating  equipment  normally  available  only  in  the  hardware 
laboratory.  The  simulation  data  base  is  constructed  to  replicate  system  conditions  and 
modes  of  operations.  During  normal  operation,  the  trainee  can  exercise  the  bladefold 
system  from  power-up  through  blade  spread.  The  setting  for  all  panel  light  configurations 
and  other  system  cues  is  normal  during  each  phase  of  the  blade-spread  cycle.  System 
faults  are  inserted  into  the  data  base  as  predefined  problems  and  in  all  instances,  are 
represented  by  single  component  failures.  During  a  training  exercise,  trainees  are  given  a 
brief  description  of  the  fault  symptoms  by  means  of  an  operator’s  complaint;  for  example, 
"Rotor  wing  head  positions,  but  the  blades  do  not  fold."  With  this  information,  the 
trainees  use  BEMTS  to  verify  the  fault  and  troubleshoot  the  system.  Trainees  use  actual 
system  technical  documentation  in  isolating  the  fault.  To  replace  a  suspected  faulty 
component,  the  trainees  simply  touch  the  replace  function  on  the  simulator  control  menu. 
They  are  allowed  to  replace  as  many  components  as  necessary  to  restore  the  system  to 
normal  operations.  However,  each  inappropriate  replacement  lowers  the  final  perfor¬ 
mance  score. 


The  Navy  Personnel  Research  and  Development  Center  (NAVPERSRANDCEN)  was 
tasked  by  NAVAIR  (Code  APC205-ON)  to  evaluate  the  BEMTS  in  the  training  setting  and 
to  establish  a  performance  baseline  that  could  later  be  used  to  measure  the  gains  that 
might  be  achieved  through  the  use  of  intelligent  systems  techniques  in  O-  and  I-level 
maintenance  training.  Data  collection  for  evaluating  the  baseline  configuration  began  in 
November  1985  following  completion  of  on-site  acceptance  testing  of  two  BEMTS  devices 
and  of  a  one-week  instructor  training  course  covering  BEMTS  operations. 

Objectives 

The  objectives  of  this  effort  were  to  (1)  determine  the  effectiveness  of  the  bladefold 
electro-mechanical  training  system  (BEMTS)  in  developing  organizational-level  main¬ 
tenance  skills  of  aviation  structural  mechanic,  hydraulic  (AM(H))  and  aviation  electrician 
(AE)  trainees  assigned  to  the  bladefold  system  portion  of  the  NAMTRAGRU  Detachment 
1069  SH-3  helicopter  hydraulic  and  electrical  systems  training  courses,  (2)  to  establish  a 
performance  baseline  for  future  testing  of  intelligent  tutoring  system  techniques  in 


connection  with  BEMTS  freeplay  simulation  capabilities,  and  (3)  to  evaluate  the  suitability 
of  BEMTS  to  the  NAMTRAGRU  training  environment. 


APPROACH 


BEMTS  Acceptance  Testing 

BEMTS  acceptance  testing  was  conducted  both  at  the  contractor  facility  and  at  the 
NAMTRAGRU  Detachment  1069  training  site.  Tests  were  conducted  at  the  training  site 
to  determine  compliance  with  installation  specifications  and  to  assist  in  evaluating  the 
suitability  issues  related  to  device  integration.  All  hardware,  software,  and  lesson 
materials  deficiencies  were  noted  and  tracked  during  the  remainder  of  the  test  and 
evaluation  period.  Instructor  comments  were  also  noted  and  tracked. 

Instructor  Training 

Following  completion  of  acceptance  testing,  NAMTRAGRU  Detachment  instructors 
attended  one-week  training  course  conducted  by  the  BEMTS  research  and  development 
team.  The  training  course  covered  BEMTS  operations,  instructor  functions,  and  student 
use.  Instructors  had  time  to  become  familiar  with  both  the  tutorial  and  freeplay  lessons 
in  the  lesson  and  simulation  repertoire.  Detachment  instructors  were  interviewed  at  the 
end  of  the  training  course  to  obtain  their  perceptions  of  the  adequacy  of  the  training 
course. 

Subjects 

Subjects  were  71  AM(H)  trainees  attending  the  SH-3  helicopter  airframe  and 
hydraulic  systems  organizational  maintenance  course  and  27  AE  trainees  attending  the 
SH-3  helicopter  electrical  systems  organizational  maintenance  course  conducted  at 
NAMTRAGRU  Detachment  1069,  NAS  North  Island.  Trainees  in  the  courses  differed  in 
years  of  Navy  service  and  experience  in  aviation  maintenance  training,  but  had  no 
previous  training  in  SH-3H  helicopter  maintenance. 

The  airframe  and  hydraulic  class  is  shorter  and  scheduled  more  frequently  during  the 
training  year  than  is  the  electrical  class.  An  average  of  just  over  two  AM(H)  classes  were 
conducted  for  each  AE  class  during  the  BEMTS  test  and  evaluation  period.  This  normal 
sequencing  of  classes  resulted  in  the  substantial  difference  in  the  total  number  of  trainees 
tested  within  each  technical  skills  category. 

Procedures 


The  SH-3  bladefold  system  represents  approximately  20  percent  of  the  curriculum  in 
both  the  airframe  and  hydraulic  systems  and  the  electrical  systems  organizational 
maintenance  training  courses.  Class  size  varies  from  six  to  eight  trainees  per  class.  Each 
class  averages  about  20  hours  of  training  time  in  the  laboratory  troubleshooting  bladefold 
system  malfunctions  on  the  SH-3  composite  trainer.  Troubleshooting  practice  on  the 
composite  trainer  is  performed  as  a  team  evaluation.  Because  it  is  considered  impractical 
to  have  more  than  four  trainees  on  the  composite  trainer  at  one  time,  classes  are 
typically  divided  into  two  or  more  teams.  The  teams  rotated  to  the  composite  trainer  at 
intervals  determined  by  the  time  required  to  complete  one  or  more  problem  scenarios 
(about  2  hours).  Teams  not  on  the  composite  trainer  are  assigned  to  independent  study 
reviewing  class  notes,  troubleshooting  manuals,  and  system  schematics.  Instructors  are 
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available  to  answer  any  questions  and  to  offer  technical  assistance  as  required  during  the 
independent  study  periods. 


During  the  BEMTS  test  and  evaluation,  students  in  the  hydraulic  and  electrical 
classes  were  divided  into  two  technically  equal  groups  of  three  or  four  trainees  per  group. 
Training  course  instructors  estimated  technical  equality  on  the  basis  of  time  in  the  Navy, 
prior  aviation  maintenance  experience,  and  relative  class  standing.  Both  groups  received 
the  normal  classroom  instruction  on  the  bladefold  system  in  their  respective  training 
courses.  The  SH-3  composite  trainer  only  (control)  group  followed  the  normal  training 
schedule  rotating  between  the  composite  trainer  and  independent  study.  The  BEMTS 
(experimental)  group  was  rotated  between  the  SH-3  composite  trainer  and  the  BEMTS 
device. 

With  two  BEMTSs  available,  each  BEMTS  group  was  divided  into  teams  of  one  or  two 
trainees  per  device  depending  on  group  size.  Each  team  was  introduced  briefly  to  the 
device  and  instructed  in  its  operations.  An  instructor  remained  nearby  to  monitor  team 
progress  and  to  assist  as  needed.  After  the  trainees  completed  the  tutorial  lessons  on 
BEMTS,  they  moved  to  the  freeplay  troubleshooting  exercises.  Instructors  selected  the 
sequence  of  troubleshooting  exercises  for  each  team.  Because  BEMTS  allows  trainees  to 
proceed  at  their  own  pace,  the  actual  number  of  freeplay  troubleshooting  problems 
completed  differed  between  teams  and  groups. 

The  following  performance  measures  were  used  to  determine  BEMTS  training 
effectiveness; 

1.  Individual  measures. 

a.  An  instructor's  rating  of  each  trainee's  performance  for  the  laboratory 
portion  of  the  course. 

b.  'I  he  score  (percent  correct)  on  the  written  bladefold  test  in  each  course. 

2.  Team  measures. 

a.  Time  for  each  team  to  complete  troubleshooting  a  system  malfunction  on 
the  composite  trainer  with  instructor  prompting. 

b.  Time  for  each  team  to  complete  troubleshooting  a  system  malfunction  on 
the  composite  trainer  without  instructor  prompting  (final  exam). 

c.  Number  of  errors  made  by  teams  during  troubleshooting  exercises. 

A  checklist  of  typical  troubleshooting  errors  (Appendix  A)  was  developed  to  assist  in 
recording  group  performance  on  the  composite  trainer  during  laboratory  exercises. 
Because  trainees  solved  problems  as  a  group  during  laboratory  sessions,  each  trainee  in 
the  group  received  the  same  solution  time  for  observed  system  troubleshooting  activity. 

Operational  Suitability  and  User  Acceptance 

The  following  items  were  monitored  to  assist  in  evaluating  the  suitability  of  the 
BEMT.S  in  the  existing  NAMTRAGRU  training  environment: 
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3. 

4. 


Physical  integration. 

Availability 

Appropriateness  for  bladefold  system  training. 
User  acceptance. 


User  acceptance  was  measured  through  interviews  and  questionnaires  administered  to 
participating  instructors  (p.  B-1)  and  trainees  in  the  BEMTS  group  (p.  B-2).  Questionnaires 
were  also  administered  to  instructors  and  trainees  (both  experimental  and  control  groups) 
to  elicit  opinions  concerning  training  simulators  (p.  B-3)  and  the  SH-3  composite  trainer 
(p.  B-4)  respectively. 

Analysis 

Given  the  small  sample  sizes  in  both  the  hydraulic  and  electrical  groups,  statistical 
methods  were  not  used  to  examine  group  differences  in  the  time-to-troubleshoot  malfunc¬ 
tions  on  the  composite  trainer  or  the  number  of  errors  made  during  troubleshooting.  For 
the  same  reason,  statistical  methods  were  not  used  to  examine  the  trainee  performance  of 
the  electrical  groups  on  bladefold  system  written  and  final  performance  tests.  Perfor¬ 
mance  difference  on  bladefold  system  written  and  performance  tests  for  the  hydraulic 
groups  were  analyzed  using  t-ratios.  User  acceptance  data  were  analyzed  using  the  mean 
ratings  of  responses  made  on  a  5-point  scale  to  statements  included  in  the  User 
Acceptance  Questionnaires  (pp.  B-1  and  B-2). 


RESULTS 


BEMTS  Acceptance  Testing 


Table  1  lists  the  problems  encountered  with  BEMTS  hardware,  software,  and  lesson 
materials  during  acceptance  testing  and  initial  student  use  as  well  as  their  causes  and  how 
they  were  corrected. 

Instructors  participating  in  acceptance  testing  were  interviewed  to  obtain  their 
initial  impressions  of  the  BEMTS  device.  Their  comments  are  summarized  below: 

1.  The  BEMTS  provides  a  safe  learning  environment. 

2.  Tutorial  lessons  seem  to  match  course  lessons. 

3.  The  BEMTS  should  allow  trainees  to  learn  procedures  step  by  step. 

4.  The  BEMTS  should  help  to  reduce  some  of  the  wear-and-tear  on  the  composite 
trainer. 

5.  The  BEMTS  should  provide  immediate  feedback  when  trainees  give  wrong 
answers. 

6.  BEMTS  operations  need  to  be  simplified  because  instructor  operations  are 
"unfriendly."  There  are  too  many  things  to  do  that,  if  omitted,  will  lead  to  later 
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Table  1 


Problems  Identified  and  Corrections  Made  During  BEMTS  Acceptance  Testing 


Problem 


Cause  and  Correction 


1.  Software  problems  prevented  operation 
of  the  procedures  following  routine 
in  the  freeplay  format. 


2.  Graphic  materials  did  not  always  appear 
on  the  image  display  of  one  BEMTS  trainer 
after  bootup. 


3.  Some  touch  points  were  missing  and 
touch  panels  were  out  of  calibration. 

4.  One  trainer  locked  during  bootup. 


5.  Several  trainer  exercises  contained 
incorrect  lesson  information  and  trouble¬ 
shooting  feedback. 

6.  Some  prompts  in  the  tutorial  lessons 
did  not  state  clearly  which  display 
to  touch  to  continue  the  lesson. 


Users  could  lock  the  trainer 
program  by  performing  certain 
steps  out  of  sequence.  The 
program  was  changed  to  prevent 
this  occurrence. 

A  hardware  difference  was  found 
to  exist  between  the  two  BEMTS 
trainers.  The  difference  concerned 
the  initial  setting  of  an  installed 
microchip.  A  software  patch  was 
written  to  properly  initialize  the 
microchips  on  bootup. 

Missing  touch  points  were  added 
and  panels  were  recalibrated. 

The  hard  disk  system  contained 
two  competing  files.  This  problem 
was  solved  by  reloading  trainer 
files  on  the  hard  disk  system. 

Lesson  data  base  was  corrected  in 
accordance  with  instructor- 
reviewer  specifications. 

Text  messages  were  added  to 
indicate  the  appropriate  display 
screen  with  active  touch  points. 


7.  Support  documentation  was  incomplete.  Upgrades  were  made  by  the 

development  contractor. 


problems.  For  example,  student  data  are  lost  if  the  floppy  diskette  is  removed  from  the 
drive  at  the  wrong  spot  in  the  program. 

7.  The  room  selected  for  BEMTS  is  too  hot  and  has  no  air  conditioning. 

8.  An  on-site  computer  expert  might  be  needed  in  the  event  the  system  fails. 

9.  Collectively,  instructors  prefer  actual  equipment  trainers  over  devices  of  the 
BEMTS  type. 
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Instructor  Training 

Instructor  comments  regarding  the  BEMTS  operations  training  course  are  summarized 
below; 

1.  Both  the  course  and  course  length  were  minimally  adequate. 

2.  The  contractor-furnished  instructor  operations  manual  was  poorly  organized  and 
difficult  to  understand. 

3.  Classroom  presentations  were  well  organized  and  cleared  up  ambiguities  in  the 
instructor  operations  manual. 

4.  The  instructor  operations  manual  should  include  simplified  procedures  to  help  the 
operator  use  and  understand  the  various  system  diskettes  that  come  with  the  BEMTS. 

Physical  Integration 

A  9-  by  18-foot  office  space  contained  the  two  BEMTSs  separated  by  a  printer  they 
both  shared,  a  standard  size  desk  for  instructor  use,  and  adequate  115  volt  60  Hz 
electrical  power,  power  outlets,  and  fluorescent  lighting  for  the  BEMTS  installation.  Air 
conditioning  was  not  available  in  the  training  space  and  fans  were  sometimes  used  to 
lower  temperatures  that  reached  as  high  as  OO^F  during  the  summer  months.  Although 
not  an  ideal  training  environment,  the  high  temperatures  did  not  appear  to  affect  student 
or  equipment  performance. 

Training  Effectiveness 

Tables  2  and  3  summarize  the  performance  data  for  the  hydraulic  classes.  The 
experimental  group  obtained  a  significantly  higher  mean  score  on  the  bladefold  system 
written  test  than  did  the  control  group.  The  difference  in  the  mean  scores  received  for 
laboratory  troubleshooting  performance  was  not  statistically  significant,  although  the 


Table  2 

Individual  Training  Effectiveness  Measures:  Hydraulic  Classes 


Mean  Score  (%) 


Item 

Control  Group 
n  =  35 

Experimental  Group 
n  =  36 

Difference 

df 

t-Test 

Bladefold  written 
test 

89.7 

93.1 

3.4 

69 

2.01* 

Laboratory  trouble¬ 
shooting 

67.7 

69.2 

1.5 

69 

.60 

Table  3 


Team  Laboratory  Performance  Measures:  Hydraulic  Classes 


Mean  Result 

Item 

Control  Group 
n  =  10 

Experimental  Group 
n  =  10 

Difference 

Time  (min)  to  troubleshoot 
malfunctions  (prompted) 

30.7 

27.4 

3.3 

Time  (min)  to  troubleshoot 
malfunctions  (Unprompted) 

43.3 

29.4 

13.9 

Number  of  troubleshooting  errors 
to  problem  solution 

.89 

.74 

.15 

experimental  group  did  obtain  a  slightly  higher  mean  score.  In  team  laboratory 
performance,  the  experimental  group  achieved  a  lower  mean  time  to  troubleshoot 
malfunctions  on  the  composite  trainer  in  both  the  instructor  prompted  and  unprompted 
categories  and  in  the  mean  number  of  errors  to  problem  solution.  Because  of  the  small 
number  of  teams  in  the  sample,  these  differences  were  not  subjected  to  statistical  tests 
of  significance. 

Tables  4  and  5  summarize  the  mean  performance  data  for  the  electrical  classes. 
Because  of  the  limited  number  of  classes  and  trainees  available  during  the  testing  period, 
individual  and  group  performance  data  were  not  subjected  to  statistical  tests  of 
significance.  However,  as  with  the  hydraulic  groups,  the  trends  observed  for  the 
electrical  experimental  group  were  all  in  a  positive  direction  regarding  BEMTS  effective¬ 
ness.  The  electrical  experimental  groups  obtained  slightly  higher  mean  scores  on  written 
and  final  performance  tests  than  did  the  control  groups.  They  also  achieved  a  lower  mean 
time  to  troubleshoot  malfunctions  on  the  composite  trainer  and  lower  mean  number  of 
errors  to  problem  solution  in  both  the  prompted  and  unprompted  categories. 


Table  4 

Individual  Training  Effectiveness  Measures:  Electrical  Classes 


Mean  Score  (%) 

Item 

Control  Group 
n  =  13 

Experimental  Group 
n  =  14 

Difference 

Bladefold  written  test 

87.4 

90.6 

3.2 

Laboratory  troubleshooting 

91.6 

93.9 

2.3 

Table  5 


Team  Laboratory  Performance  Measures:  Electrical  Classes 


Mean  Result 


Item 

Control  Group 
n  =  4 

Experimental  Group 
n  =  4 

Difference 

Time  (min)  to  troubleshoot 
malfunctions  (prompted) 

46.6 

25.9 

20.7 

Time  (min)  to  troubleshoot 
malfunctions  (unprompted) 

44.0 

19.8 

24.2 

Number  of  troubleshooting  errors 
to  problem  solution 

1.6 

.55 

1.05 

Performance  data  within  each  measurement  category  show  a  consistent  positive 
trend  for  the  experimental  groups.  The  most  notable  difference  between  the  experimen¬ 
tal  and  control  groups  was  in  the  time  required  to  complete  troubleshooting  on  the 
composite  trainer  without  prompting.  Mean  differences  of  13.9  and  24.2  minutes  to 
problem  solution  were  recorded  for  the  hydraulic  and  electrical  classes  respectively. 
These  differences  represent  an  average  reduction  of  43  percent  in  the  time  required  to 
complete  a  typical  troubleshooting  problem  on  the  composite  trainer  for  both  experimen¬ 
tal  groups.  The  experimental  group  in  the  electrical  classes  recorded  similar  percentage 
gains  in  the  other  two  laboratory  performance  categories. 

Equipment  Availability 

Table  6  shows  the  downtime  accumulated  for  both  the  SH-3  composite  trainer  and  the 
BEMTS  during  the  testing  period.  The  composite  trainer  was  down  for  8  days  awaiting 


Table  6 

Trainer  Downtime  for  Preventive  and  Corrective  Maintenance 


Downtime 


Composite  BEMTS  Difference 

Trainer  (2  devices) 


Preventive  main- 


tenance 

4.5  hours 

4.2 

hours 

.3 

hours 

Corrective  main- 

tenance 

8  days 

1 

day 

7 

days 

2  hours 

10.7 

hours 

8.7 

hours 

^Awaiting  replacement  part. 


delivery  of  a  check  valve  that  was  not  available  locally.  The  2  hours  recorded  for 
corrective  maintenance  involved  the  replacement  of  the  one-way  check  valve  and  an 
auxiliary  pressure  switch. 

BEMTS  downtime  consisted  1  day  to  replace  a  faulty  high  resolution  display  monitor 
and  approximately  1 1  hours  for  corrective  maintenance.  BEMTS  corrective  maintenance 
involved  the  repair  of  several  hardware  and  software  problems  that  instructors  initially 
identified  during  acceptance  testing  (Table  1)  but  in  many  instances  took  several  weeks  to 
repair.  Because  the  test  and  evaluation  effort  actually  began  before  all  identified 
problems  had  been  resolved,  BEMTS  use  by  one  early  hydraulic  class  had  to  be  cancelled. 
Of  the  remaining  problems,  over  70  percent  involved  some  form  of  instructor-operator 
error.  Instructors  often  overlooked  a  procedural  step  in  preparing  student  floppy  diskettes 
correctly  or  in  removing  them  from  the  computer.  These  oversights  would  cause  the 
BEMTS  programs  to  stop  and  often  resulted  in  the  loss  of  recorded  student  data. 
Corrective  actions  involved  turning  system  power  off  and  on  or  depressing  the  reset  button 
to  reboot  the  system.  A  system  modification  was  installed  in  the  third  month  of  testing 
which  eliminated  the  use  of  student  floppy  diskettes.  This  modification  resolved  many  of 
the  observed  instructor  difficulties  in  preparing  BEMTS  for  student  use  and  the  computer 
program  stops  associated  with  floppy  diskette  operations. 


Downtime  for  preventive  maintenance  was  approximately  equal  for  the  BEMTS  and 
composite  trainers  and  did  not  interfere  with  on-going  training. 


User  Acceptance 


Tables  7  and  8  present  the  mean  responses  based  on  a  scale  of  1  to  5  to  items 
included  in  student  (p.  B-2)  and  instructor  (p.  B-l)  user  acceptance  questionnaries.  In 
general,  trainees  in  the  experimental  groups  indicated  that  the  BEMTS  was  easy  to  use, 
helped  them  to  understand  the  course  materials,  helped  them  to  learn  how  to  troubleshoot 
the  bladefold  system,  and  measured  their  troubleshooting  ability  accurately.  Instructors 
in  both  training  courses  indicated  that  BEMTS  lessons  were  compatible  and  consistent 
with  course  objectives  and  that  the  training  received  on  BEMTS  was  a  useful  supplement 
to  that  received  on  the  composite  trainer.  According  to  the  instructors,  BEMTS  student 
data  collection  features  of  the  BEMTS  were  only  marginally  effective  in  monitoring 
student  performance  and  BEMTS  lesson  authoring  software  did  not  offer  an  easy  method 
for  developing  new  lessons.  Instructors  overwhelmingly  agreed  that  the  use  of  BEMTS 
added  to  their  workload. 


Follow-up  interviews  with  the  instructors  tended  to  validate  the  responses  received 
on  the  user  acceptance  questionnaires.  Instructors  in  both  the  hydraulic  and  electrical 
system  courses  considered  the  BEMTS  appropriate  for  use  in  bladefold  system  training. 
They  also  stated  that  the  knowledge  and  skills  taught  using  the  tutorial  and  troubleshoot¬ 
ing  lessons  were  relevant  and  consistent  with  stated  course  objectives  and  compatible 
with  those  taught  on  the  composite  trainer.  According  to  most  of  the  instructors,  BEMTS 
incorporates  useful  training  materials  and  approaches  that  could  be  expected  to  increase 
the  quality  of  training  in  NAMTRAGRU  Detachments.  However,  they  consistently 
pointed  out  that  they  did  not  believe  BEMTS  could  be  used  effectively  alone.  In  their 
opinion,  BEMTS  should  only  be  used  to  complement  existing  training.  Instructors  agreed 
that  teaching  safety  precautions  was  the  weakest  area  in  BEMTS  training.  They 
commented  that  safety  related  feedback  is  either  unrealistic  or  nonexistent. 


Table  7 


Student  Acceptance  of  BEMTS 


Student  Mean  Responses 


Item 

Hydraulic 

Group 

Electrical 

Group 

I. 

The  tutorial  lessons  helped  me  to  learn  about 
the  bladefold  system. 

3.67 

4.53 

2. 

The  tutorial  lessons  were  too  long. 

2.14 

1.94 

3. 

The  tests  at  the  end  of  the  tutorial  lessons 
were  too  difficult. 

1.41 

1.67 

4. 

The  problems  helped  me  learn  to  troubleshoot. 

3.89 

4.71 

5. 

The  problems  were  too  difficult  to  solve. 

1.81 

1.53 

6. 

The  problems  took  too  long  to  solve. 

2.31 

1.94 

7. 

A  variety  of  troubleshooting  problems  was 
presented. 

4.31 

4.82 

8. 

The  computer  was  easy  to  use. 

4.03 

4.76 

9. 

There  was  enough  training  time  on  the  computer. 

3.61 

3.65 

10. 

More  time  should  be  spent  doing  troubleshooting 
problems  on  the  computer. 

3.14 

3.47 

11. 

Training  time  on  the  computer  was  often  wasted. 

1.29 

2.25 

12. 

Computer  breakdowns  interfered  with  training. 

2.47 

1.76 

13. 

The  computer  helped  me  to  understand  the  actual 
equipment. 

3.67 

4.65 

14. 

My  performance  on  the  computer  was  an  accurate 
measure  of  my  troubleshooting  ability. 

3.08 

3.94 

Table  8 


Instructor  Acceptance  of  BEMTS 


Instructor  Mean  Responses- 

Hydraulic 

Electrical 

Item 

Course 

Course 

1. 

The  tutorial  lessons  are  well  designed  for 
bladefold  instruction. 

3.90 

4.50 

2. 

The  tutorial  lessons  meet  course  objectives. 

3.80 

4.50 

3. 

The  troubleshooting  lessons  are  well  designed 
for  bladefold  instruction. 

3.20 

4.50 

4. 

The  troubleshooting  lessons  meet  course 
objectives 

3.50 

4.00 

5. 

The  computer  trainer  provides  a  useful  supple¬ 
ment  to  the  actual  equipment. 

3.60 

4.50 

6. 

The  authoring  software  is  easy  to  use. 

2.70 

4.25 

7. 

The  authoring  software  provides  an  effective 
method  of  developing  new  lessons. 

2.50 

2.50 

8. 

The  record  keeping  features  of  the  computer 
trainer  simplify  the  task  of  individual 
student  management. 

3.30 

3.00 

9. 

The  addition  of  the  computer  trainer  does  add 
significantly  to  the  instructional  workload. 

1.40 

1.50 

10. 

Student  data  produced  by  the  computer  trainer 
are  helpful  in  student  management. 

2.80 

2.75 

11. 

Student  data  produced  by  the  computer  are  useful 
in  assessing  student  abilities. 

2.80 

3.25 

^Responses  based  on  a  scale  of  1  to  5  where  1  =  strongly  disagree  and  5  =  strongly  agree. 


A  separate  questionnaire  was  given  to  instructors  to  obtain  their  opinions  concerning 
the  general  use  of  simulators  in  training  (p.  B-3).  Table  9  presents  the  instructors'  mean 
responses  (on  a  scale  of  1  to  5)  to  items  comparing  the  use  of  simulators  to  actual 
equipment  in  training.  Consistent  with  the  ratings  received  on  the  BEMTS  user 
acceptance  questionnaire,  instructors  in  both  the  hydraulic  and  electrical  courses  agreed 
that  simulators  were  a  good  idea,  monitored  student  performance  better,  helped  in 
meeting  course  training  objectives,  and  provided  training  in  less  time  and  at  lower  cost 
than  did  actual  equipment.  They  were  less  inclined  to  agree  that  simulators  were  more 
reliable  and  easier  to  maintain  or  use  than  actual  equipment.  The  two  instructor  groups 
differed  in  their  answers  to  questions  relating  to  training  effectiveness  and  simulator 
fidelity.  The  electrical  instructors  did  not  believe  that  simulators  could  be  as  effective  as 
the  actual  equipment  or  that  high  simulator  fidelity  was  required  to  meet  training  needs. 


Table  9 


Instructor  Evaluation  of  Training  Simulators 


Instructor  Mean  Responses^ 

Based  on  my  knowledge  and  experience,  Hydraulic  Electrical 

1  think  that  training  simulators:  Course  Course 


1. 

are  a  good  idea. 

4.67 

4.60 

2. 

can  be  as  effective  as  actual  equipment. 

3.44 

2.20 

3. 

must  look  like  actual  equipment. 

4.67 

2.80 

4. 

can  provide  training  at  a  lower  cost  than 
actual  equipment. 

3.33 

3.40 

5. 

can  provide  training  in  less  time  than 
actual  equipment. 

3.33 

3.20 

6. 

can  present  more  complex  training  problems 
than  actual  equipment. 

3.56 

3.60 

7. 

are  more  reliable  than  actual  equipment. 

2.56 

2.60 

8. 

can  be  maintained  as  easily  as  actual  equip¬ 
ment. 

2.89 

2.00 

9. 

teach  safety  better  than  actual  equipment. 

3.22 

1.40 

10. 

provide  for  better  monitoring  of  student 
performance  than  actual  equipment. 

3.78 

4.00 

11. 

are  as  easy  for  instructors  to  use  as 
actual  equipment. 

3.22 

2.80 

12. 

are  as  easy  for  students  to  use  as  actual 
equipment. 

3.22 

3.60 

13. 

are  something  I  would  use  as  an  integral  part 
of  the  courses  that  I  teach. 

3.89 

4.20 

Responses  based  on  a  scale  of  1  to  5  where  1  =  strongly  disagree  and  5  =  strongly  agree. 


The  hydraulic  instructors  indicated  that  simulators  could  be  as  effective  as  actual 
equipment  but  required  high  actual  equipment  fidelity.  The  two  groups  also  disagreed  on 
the  ability  of  simulators  to  teach  safety  procedures. 

A  third  questionnaire  (p.  B-4)  was  given  to  all  students  to  obtain  their  opinions  of 
training  performed  on  the  SH-3  composite  trainer.  Table  10  presents  the  students'  mean 
responses  (on  a  scale  of  1  to  5)  to  items  relating  to  the  composite  trainer  effectiveness 
and  ease  of  use.  Both  experimental  and  control  groups  agreed  that  training  with  the 
composite  trainer  was  effective  in  helping  to  learn  troubleshooting  procedures,  that 
training  time  was  well  spent,  and  that  their  performance  on  the  composite  trainer 
measured  their  system  troubleshooting  ability  accurately. 
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Table  10 


Student  Evaluation  of  SH-3  Helicopter  Composite  Trainer 


/draulic  Course 


Electrical  Course 


Item 

Experimental 

Group 

Control 

Group 

Experimental 

Group 

Control 

Group 

1. 

I  can  follow  the  cockpit  checklist 
without  making  mistakes. 

4.64 

4.63 

4.47 

4.85 

2. 

1  can  use  the  schematics  to  trouble¬ 
shoot  the  bladefold  system. 

4.44 

4.34 

4.46 

4.71 

3. 

I  know  the  names  and  locations  of 
the  bladefold  components. 

4.00 

3.74 

3.94 

3.92 

4. 

I  understand  how  to  troubleshoot 
the  bladefold  components. 

4.39 

4.37 

4.59 

4.77 

5. 

I  feel  comfortable  working  on  the 
bladefold  system. 

4.36 

4.00 

4.82 

4.38 

6. 

There  was  enough  training  time 
on  the  equipment. 

3.94 

4.20 

4.23 

3.88 

7. 

Training  time  on  the  equipment 
was  often  wasted. 

1.39 

1.37 

1.29 

1.15 

8. 

A  variety  of  troubleshooting 
problems  was  provided. 

4.75 

4.86 

4.82 

4.85 

9. 

The  problems  helped  me  learn  to 
troubleshoot. 

4.78 

4.66 

4.88 

4.92 

10. 

The  problems  were  too  difficult 
to  solve. 

1.81 

1.40 

1.59 

1.38 

11. 

The  problems  took  too  long  to 
solve. 

2.08 

1.91 

1.65 

1.85 

12. 

Equipment  breakdowns  interfered 
with  training. 

1.31 

1.31 

1.59 

1.38 

13. 

More  time  should  be  spent  doing 
traoubleshooting  problems  on  the 
actual  equipment. 

3.39 

3.51 

3.71 

3.54 

14. 

My  performance  on  the  actual 
equipent  was  an  accurate  measure 
of  my  troubleshooting  ability. 

3.86 

3.97 

4.15 

4.29 

a 


Responses  based  on  a  scale  of  1  to  5  where  1  =  strongly  disagree  and  5  =  strongly  agree. 
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DISCUSSION 


Simply  stated,  the  questions  posed  for  this  evaluation  were  (1)  will  technician 
trainees  demonstrate  better  system  troubleshooting  performance  on  the  SH-3  helicopter 
bladefold  system  composite  trainer  as  a  result  of  using  BEMTS,  and  (2)  is  this  technology 
suited  to  the  existing  training  environment?  The  answer  to  the  first  question  is  a 
qualified  yes.  Only  in  the  hydraulics  classes  did  the  samples  contain  enough  trainees  to 
perform  a  statistical  analysis  on  written  and  laboratory  performance  test  scores. 
Although  the  experimental  group  mean  score  was  higher  for  both  of  these  measures,  only 
the  difference  in  the  written  test  scores  was  found  to  be  statistically  significant.  Since 
the  experimental  and  control  groups  in  both  the  hydraulic  and  electrical  classes  scored 
well  within  a  range  to  assume  mastery  of  the  course  materials,  the  recorded  superiority  of 
the  experimental  groups  on  these  measures  (Tables  2  and  4)  is  not  overwhelming. 

The  differences  in  mean  scores  for  time  to  troubleshoot  a  malfunction  on  the 
composite  trainer  and  the  number  of  errors  to  problem  solution  (Tables  3  and  5)  are  much 
more  impressive.  In  the  hydraulics  classes,  the  experimental  group  averaged  13.9  minutes 
less  to  troubleshoot  malfunctions  without  prompting  than  did  the  control  group.  In  the 
electrical  classes,  the  experimental  group  averaged  24.2  minutes  less  on  this  measure. 
The  electrical  experimental  group  also  averaged  20.7  minutes  less  to  troubleshoot 
malfunctions  with  prompting  and  averaged  one  less  error  to  problem  solution  where  the 
control  group  averaged  less  than  two  errors.  While  the  mean  scores  for  the  hydraulic 
experimental  group  did  not  equal  the  magnitude  of  difference  recorded  for  the  electrical 
experimental  group  in  these  latter  measures,  they  were  consistently  higher  than  the 
control  group  scores.  Given  the  small  number  of  trainees  in  the  test  sample,  the 
reliability  of  the  positive  trend  may  be  questioned.  However,  BEMTS  performance  data 
appear  to  replicate  the  results  of  similar  tests  of  the  simulation  and  training  approach 
(Cicchinelli,  1984;  Rigney,  1980)  and  continue  to  be  quite  encouraging. 

The  simulation  and  training  techniques  employed  in  BEMTS  were  originally  developed 
using  electronic  equipment  for  concept  evaluation.  The  underlying  principles  involve 
techniques  to  separately  develop  and  exercise  the  cognitive  skills  associated  with  system 
diagnosis  and  troubleshooting.  For  electronic  systems,  symptom  diagnosis  most  often 
involves  the  use  of  test  equipment.  The  SH-3  helicopter  bladefold  system  is  an  electro¬ 
mechanical  system  that  electrically  requires  many  of  the  same  circuit  diagnosis  and 
testing  skills  employed  in  troubleshooting  electronic  systems.  On  the  hydraulic  side, 
system  diagnosis  and  testing  are  very  different.  Hydraulic  system  failures  are  often 
analyzed  and  isolated  by  observing  the  tensioning  effects  on  hydraulic  lines,  the  stability 
of  the  blade  spread/fold  cycle,  or  by  the  occurrence  of  auditory  and  visual  anomalies  in 
mechanical  sequences.  BEMTS  did  not  include  capabilities  for  developing  and  exercising 
auditory  or  visual  skills.  The  omission  of  these  capabilities  from  the  simulation  could  very 
likely  have  contributed  to  the  differences  observed  in  laboratory  troubleshooting  perfor¬ 
mance  between  the  hydraulic  and  electrical  experimental  groups. 

The  data  collected  in  this  test  and  evaluation  provide  no  easy  answer  to  the  question 
of  how  well  the  technology  is  suited  to  the  NAMTRAGRU  training  environment.  The 
BEMTS  was  designed  as  a  system  prototype  to  test  the  simulation  and  system  trouble¬ 
shooting  practice  concept  in  SH-3  helicopter  maintenance  training  and  for  future  testing 
of  intelligent  tutoring  system  applications  in  the  NAMTRAGRU  training  environment. 
Acceptance  testing  of  the  BEMTS  prior  to  installation  in  the  NAMTRAGRU  facilities 
uncovered  several  minor  equipment  and  training  material  problems  that  were  easily 
solved.  However,  acceptance  testing  did  not  subject  the  BEMTS  to  the  same  level  of 
scrutiny  that  accompanied  implementation  and  student  use.  As  with  any  new  technology. 
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instructors  need  time  to  become  familiar  with  both  BEMTS  operations  and  the  content  and 
format  of  the  various  instructional  paradigms  employed.  Many  of  the  problems  noted 
occurred  within  the  first  three  months  of  testing.  Many  of  these  problems  were 
attributed  to  poor  touch  panel  calibration,  a  general  disagreement  by  instructors 
concerning  the  syntax  or  terminology  used  to  explicate  system  and  component  functions, 
or  operator  error.  Other  problems  involved  intermittent  spurious  operation  of  the 
computer  system  and/or  the  controlling  software.  Since  the  BEMTS  research  and 
development  team  could  not  replicate  many  of  the  difficulties,  not  all  of  these 
intermittent  problems  were  isolated  or  resolved. 

One  of  the  most  frequently  cited  BEMTS  problems  and  the  one  which  contributed 
most  to  negative  responses  by  instructors  was  associated  with  the  use  of  floppy  diskettes. 
When  delivered  by  the  development  contractor,  BEMTS  required  that  the  instructors 
initialize  a  separate  floppy  diskette  for  each  student  and/or  pair  of  students  assigned  a 
BEMTS  device.  The  initialization  process  required  instructors  to  create  a  student  diskette 
with  a  student  number,  name,  and  the  trainer  exercises  in  which  the  student  would 
participate  (tutorial  and/or  freeplay).  Instructors  received  no  feedback  to  verify  that  the 
diskette  was  properly  initialized.  The  procedures  also  required  that  floppy  diskettes  be 
inserted  and  removed  only  at  specific  points  during  the  boot-up  and  sign-off  process.  If  a 
diskette  was  not  properly  initialized,  the  computer  system  would  lock  or  perform  random 
and  unpredictable  operations.  If  the  diskette  was  removed  before  the  training  program 
menu  was  displayed,  student  performance  data  would  be  lost  and  the  computer  would  lose 
synchronization.  Both  problems  required  that  the  instructor  reboot  the  system  and  start 
at  the  beginning  of  the  lesson  selection  process.  After  the  first  four  classes  had 
completed  testing,  the  BEMTS  research  team  installed  a  software  modification  that 
included  a  menu  driven  selection  routine  to  replace  the  procedures  used  with  the  original 
instructor  utility  program.  The  menu  selection  routine  eliminated  most  of  the  keyboard 
entries,  prompted  all  required  instructor  actions,  and  allowed  instructors  to  create 
student  files  directly  onto  the  computer  hard  disk  system,  thereby  eliminating  the  use  of 
student  floppy  diskettes  in  the  training  mode.  This  modification  resolved  many  of  the 
instructor  difficulties  associated  with  using  BEMTS  and  also  many  of  the  instructor 
complaints  relating  to  the  computer  system. 

The  bladefold  system  represents  about  20  percent  of  the  SH-3  helicopter  training 
course  and  the  BEMTS  was  actually  used  in  less  than  20  percent  of  available  bladefold 
system  training  time.  The  BEMTS  test  and  evaluation  increased  the  instructors  normal 
workload  by  approximately  the  number  of  hours  the  instructors  spent  monitoring  trainees 
using  BEMTS.  It  is  not  clear  if  these  additional  hours  were  imposed  because  of  BEMTS 
requirements  or  NAMTRAGRU  policy  to  have  at  least  one  instructor  in  all  spaces  where 
electronic  equipment  is  being  operated.  Available  data  do  not  reveal  whether  BEMTS 
would  actually  impose  significant  staffing  changes  if  its  use  was  expanded  to  include  other 
applicable  portions  of  the  SH-3  helicopter  course  and  it  was  installed  in  a  space  normally 
manned  by  staff  instructors. 

Although  the  contractor  training  course  briefly  discussed  simulation  and  lesson 
authoring,  NAMTRAGRU  instructors  were  not  required  to  author  or  modify  any  lessons 
during  the  evaluation.  Until  the  new  instructor  utility  menu  system  was  modified,  most 
instructors  found  the  tasks  associated  with  preparing  the  BEMTS  for  student  use  difficult 
to  master.  Simulation  authoring,  lesson  development  and  modification  are  much  more 
complicated.  Currently  they  require  an  in-depth  understanding  of  the  computer  operating 
system,  the  use  of  computer  text  and  graphics  editors,  computer  filing  systems,  how  the 
simulation  data  base  is  structured,  instructional  design  methodologies  as  well  as  opera¬ 
tional  systems  knowledge.  This  combination  of  skills  and  knowledge  is  not  commonly 
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available  in  most  training  commands  and  is  not  easily  passed  down  to  replacement 
instructors.  These  tasks  for  those  commands  that  are  not  currently  using  various  forms  of 
the  technology  are  additive  and  will  definitely  impose  on  existing  instructor  workloads. 
They  will  also  require  a  minimum  of  40  hours  of  training  to  allow  instructors  to  become 
familiar  with  both  the  simulation  and  tutorial  lesson  data  file  formats  and  associated 
authoring  procedures. 


CONCLUSIONS 

Although  the  results  of  this  test  and  evaluation  are  inconclusive  because  of  the 
limited  number  of  students  in  the  test  sample,  they  suggest  that  BEMTS  is  an  effective 
supplement  to  SH-3  helicopter  bladefold  system  hydraulic  and  electrical  training  currently 
conducted  on  the  SH-3  helicopter  composite  trainer  at  NAMTRAGRU  Detachment  1069. 
Specifically,  the  findings  indicate  that: 

1.  The  use  of  BEMTS  reduces  mean  troubleshooting  time  and  mean  errors  to 
problem  solution  for  both  AM(H)  and  AE  technician  trainees  performing  exercises  on  the 
SH-3  helicopter  composite  trainer. 

2.  The  use  of  BEMTS  increases  mean  test  scores  on  written  tests  of  bladefold 
systems  knowledge  for  both  AM(H)  and  AE  technician  trainees. 

3.  The  data  collected  in  this  effort  establish  an  adequate  performance  base  from 
which  to  measure  and  assess  the  effects  of  on-line  intelligent  tutoring  techniques  used  in 
conjunction  with  BEMTS  freeplay  troubleshooting  exercises. 

The  following  conclusions  address  the  continued  use  of  BEMTS  in  NAMTRAGRU 
maintenance  training  courses  and  future  testing  for  intelligent  Tutoring  system  applica¬ 
tions: 


1.  The  BEMTS  did  not  adequately  simulate  or  provide  the  kind  of  visual  and 
auditory  fault  symptoms  that  AM(H)s  use  to  diagnose  and  troubleshoot  the  bladefold 
system.  While  these  features  are  within  the  capabilities  of  existing  hardware  and 
software  configuration,  modifications  to  the  existing  simulation  data  base  and  videodisc 
are  required  to  determine  the  cost  and  performance  impact  resulting  from  these 
simulation  enhancements. 

2.  Because  it  was  not  possible  to  accurately  determine  whether  the  additional  hours 
spent  by  instructors  in  monitoring  BEMTS  student  progress  were  device  or  regulation 
driven,  changes  to  existing  NAMTRAGRU  detachment  staffing  requirements  as  a  result  of 
using  BEMTS  could  not  be  determined. 


3.  BEMTS  simulation  and  lesson  authoring  and  materials  upgrade  tasks  r  e  not 
within  the  present  or  projected  capabilities  of  NAMTRAGRU  instructors.  Theref 'tp,  all 
changes  and  modifications  to  existing  lesson  materials  and  the  development  oi  nf 
materials  will  require  outside  support  or  additional  training  for  instructors. 

4.  The  BEMTS  prototype  is  not  consistent  with  recent  trends  toward  standardiza¬ 
tion  on  PC  compatible  hardware. 
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RECOMMENDATIONS 

1.  Given  the  positive  direction  of  the  results  of  using  BEMTS  as  a  supplement  to  the 
present  hands-on  equipment  training  approach  in  SH-3  bladefold  system  training,  NAVAIR 
and  NAMTRAGRU  should  continue  to  support  a  policy  to  refine  and  institutionalize  the 
technology  base  established  in  the  BEMTS. 

2.  The  BEMTS  hardware  components  that  are  not  commercially  available  should  be 
replaced  with  PC  compatible  devices.  Future  testing  should  include  an  investigation  of 
BEMTS  software  transportability  to  other  aviation  maintenance  training  courses  where 
similar  low-cost  training  systems  are  being  planned  or  used. 

3.  The  BEMTS  simulation  and  tutorial  lesson  data  bases  should  be  expanded  to 
include  other  applicable  portions  of  the  SH-3  helicopter  maintenance  training  courses  and 
additional  tests  should  be  conducted  to  measure  the  actual  impact  of  BEMTS  on  instructor 
workloads  under  full  curriculum  implementation  conditions. 

4.  The  display  techniques  incorporated  in  the  BEMTS  freeplay  and  tutorial  para¬ 
digms  should  be  expanded  to  implement  the  visual  and  auditory  cues  AM(H)  technicians 
use  to  diagnose  hydraulic  systems  failures  and  tests  should  be  designed  to  measure  the 
applicability  and  effectiveness  of  these  simulation  and  training  enhancements. 

5.  Responsibility  for  BEMTS  simulation  and  lesson  authoring  should  not  be  assigned 
to  the  user  command  unless  such  tasking  includes  appropriate  training  and  corresponding 
augmentations  in  instructor  staffing  plans. 
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APPENDIX  B 


USER  ACCEPTANCE  QUESTIONNAIRES 

Instructor  Evaluation  of  BEMTS  . . 

Student  Evaluation  of  BEMTS  . 

Instructor  Evaluation  of  Training  Simulators  . 

Student  Evaluation  of  Actual  Equipment  Trainer 

(SH-3  Composite  Trainer) . . 


Om  ACGIRAIICI  QDUTXOMKAXU 


XNmUCTOlt  IVALUAT20N  OF  BEMT8 


TIm  follMring  4«terik«  tk*  IBMtt.  Circl*  th«  Auabtr  th«t 

•xBtAAttt  how  ouch  yoo  oiroo  or  41t«gro«  with  ooeh  ttatoMot. 


Dlsagraa 

Strongly 

Agrac 

Strongly 

1. 

The  tutorial  laaaona  are  %rell 
deaigned  for  bladefold  Inatruetlon. 

1 

2 

3 

A 

5 

2. 

The  tutorial  leaaons  acet  courae 
objectivea. 

1 

2 

3 

A 

5 

3. 

The  t rouble ehootlng  Icssods  arc  well 
designed  for  bladcfold  Instruction. 

1 

2 

3 

A 

5 

A. 

The  troubleshooting  lessons  aeet 
course  objectives. 

1 

2 

3 

A 

5 

3. 

The  coiBputer  trainer  provides  a  useful 
aupplcaent  to  ihe  actual  e<]Qlpnent . 

1 

2 

3 

A 

S 

6. 

The  authoring  software. Is  easy  to  use. 

1 

2 

3 

A 

5 

7. 

The  authoring  software  provides  sn 
effective  aethod  of  developing  new 
lessons. 

1 

2 

3 

A 

5 

8. 

The  record  keeping  features  of  the 
coaputcr  trainer  slapllfy  the  task 
of  Individual  student  aanageaent. 

1 

2 

3 

A 

5 

9. 

The  addition  of  the  coaputcr  trainer 
does  not  add  significantly  to  the 
Instructional  workload. 

1 

2 

3 

A 

5 

10 

.  Student  data  produced  by  the  coaputcr 
trainer  arc  helpful  in  student  aanageaent. 

1 

2 

3 

A 

5 

11 

.  Student  data  produced  by  the  coaputcr 
arc  useful  in  assessing  student  abilities. 

1 

2 

3 

A 

5 
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Th«  followlni  tttttMntt  dttcrlb*  optrUnett  with  th«  IIKTt  trainer. 
Circla  th«  aiabar  that  beat  axpraaaaa  hev  aueh  you  agraa  with  each 
atataaant.  Circla  1  if  you  do  not  agraa  at  all  with  tha  atatanant) 
circla  5  if  you  agraa  vary  much  with  tha  atataaant. 


1.  The  tutorial  Icaaona  helped  ne  to  learn 
about  the  bladefold  ayeten. 

2.  The  tutorial  leaaojga  were  too  long. 

3.  The  testa  at  the  end  of  the  turorlal 
lessons  were  too  difficult. 

4.  The  problcas  helped  «e  leem  to 
troubleshoot . 


ho  Not  Agree 
Agree  Very 
At  All  Much 

1  2  3  4  5 


2  3  4  5 

2  3  4  5 

2  3  4  5 


3.  The  prob leas  were  too  difficult  to  solve. 


2  3  4  5 


6.  The  problcas  took  too  long  to  solve. 


2  3  4  5 


7.  A  variety  of  troubleshooting  problesis  was 
presented. 


2  3  4  5 


8.  The  computer  was  easy  to  use. 

9.  There  was  enough  training  tlae  on  the 
computer. 

10.  More  time  should  be  spent  doing  troubleshooting 
problems  on  the  computer. 

11.  Training  lime  on  the  computer  was  often  wasted. 


2  3  4  5 

2  3  4  5 


12.  Computer 'breakdowns  Interfered  with  training. 

13.  The  computer  helped  me,  to  understand  the 
actual  equipment. 

14.  My  performance  on  the  computer  was  an  accurate 
measure  of  my  troubleshooting  ability. 


2  3  4  5 

2  3  4  5 

2  3  4  5 
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D8IR  ACCtFTAMCI  OUlfTZONNAZRI 


TORS 


The  following  statements  describe  opinions  about  training  simulators. 
Circle  the  number  that  best  expresses  how  much  you  agree  or  disagree 
with  each  statement. 


Disagree 

Strongly 


Based  on  my  knowledge  and  experience, 
I  think  that  training  simulators: 


Agree 

Strongly 


1.  are  a  good  Idea. 

2.  can  be  as  effective  as  actual  equipment. 

3.  must  look  like  actual  equipment. 

4.  can  provide  training  at  a  lower  cost  than 
actual  equipment. 

5.  can  provide  training  in  less  time  than 
actual  equipment. 

6.  can  present  more  complex  training  problems 
than  actual  equipment. 

7.  are  more  reliable  than  actual  equipment. 

8.  can  be  maintained  as  easily  as  actual 
equipment. 

9.  teach  safety  better  than  actual  equipment. 

10.  provide  for  better  monitoring  of  student 
performance  than  actual  equipment. 

11.  are  as  easy  for  instructors  to  use  as  actual 
equipment . 

12.  are  as  easy  for  students  to  use  as  actual 
equipment . 

13.  are  something  I  would  use  as  an  integral 
part  of  the  courses  that  I  teach. 


USER  ACCEPTANCE  QUEtTXONNAXEE 
STUDENT  EVALUATION  OF  ACTUAL  EQUIPMENT  TRAINEE 

Tht  followlni  acAtcMnci  d«tcrlb«  overall  axparlancaa  In  actual  aqulpnant 
trainar.  Circle  the  number  that  beat  expreaaea  how  much  you  agree  with 
each  statement.  Circle  1  if  you  do  not  agree  at  all  with  the  statement; 
circle  5  if  you  agree  very  much  with  the  atatement. 

Do  Not  Agree  Agree  Very 

At  All  Much 

1.  There  was  enough  training  time  on  the  1  2  3  4  5 

equipment . 

2.  Training  time  on  the  equipment  was  often  1  2  3  4  5 

wasted. 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 
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3.  A  variety  of  troubleshooting  problems 
was  provided. 

4.  The  problems  helped  me  learn  to 
troubleshoot. 

5.  The  problems  %rere  too  difficult  to  solve. 

6.  The  problems  took. too  long  to  solve. 

7.  Equipment  breakdowns  interfered  with 
training. 

8.  More  time  should  be  spent  doing  trouble¬ 
shooting  problems  on  the  actual  equipment. 

9.  My  performance  on  the  actual  equipment  was 
an  accurate  measure  of  my  troubleshooting 
ability. 
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